6th Data Science Symposium Abstracts

The Data Science Symposium at Haus der Wissenschaft on 8/9 November 2021 in Bremen was the 6th Symposium in this series since 2017

Newsletter of the Digital Earth Project Contributions of the Alfred Wegener Institute to Digital Earth

As an important technical pillar of Digital Earth AWI computing centre provides data management and cloud processing services to the project partners. We develop project specific extensions to the AWI data flow framework O2A (Observation to Archive). Sensor registration in O2A will support a flexible handling of sensors and their metadata, e.g. for the Digital Earth showcases, methane and soil moisture measurements are in focus for smart monitoring designs and for the access to data in near real time (NRT). Furthermore, data exploration is supported by a rasterdata manager service that can be easily coupled in user ́s data workflows with other data sources, like NRT sensor data. In the following we give more details on O2A, its components and concept

Automatic data quality control for understanding extreme climate event

The understanding of extreme events strongly depends on knowledge gained from data. Data integration of mul-tiple sources, scales and earth compartments is the fo-cus of the project Digital Earth, which also join efforts on the quality control of data. Automatic quality control is embedded in the ingest component of the O2A, the ob-servation-to-archive data flow framework of the Alfred-Wegener-Institute. In that framework, the O2A-Sensor provides observation properties to the O2A-Ingest, which delivers quality-flagged data to the O2A-dash-board. The automatic quality control currently follows a procedural approach, where modules are included to implement formulations found in the literature and other operational observatory networks. A set of plausibility tests including range, spike and gradient tests are cur-rently operational. The automatic quality control scans the ingesting data in near-real-time (NRT) format, builds a table of devices, and search - either by absolute or derivative values - for correctness and validity of obser-vations. The availability of observation properties, for in-stance tests parameters like physical or operation ranges, triggers the automatic quality control, which in turn iterates through the table of devices to set the qual-ity flag for each sample and observation. To date, the quality flags in use are sequential and qualitative, i.e. it describes a level of quality in the data. A new flagging system is under development to include a descriptive characteristic that will comprise technical and user inter-pretation. Within Digital Earth, data on flood and drought events along the Elbe River and methane emissions in the North Sea are to be reviewed using automatic qual-ity control. Fast and scalable automatic quality control will disentangle uncertainty raised by quality issues and thus improve our understanding of extreme events in those cases

Permafrost-related research data - their accessibility, visualization, and publication using GIS and WebGIS technology

Permafrost regions are highly sensitive to climate changes. The monitoring of key variables and identification of relevant-processes is of topmost importance in these environments. ESA DUE GlobPermafrost (www.globpermafrost.info) provides a remote sensing data service for permafrost research and applications. This service was extended by permafrost modelling (time series), implemented in the new ESA CCI+ Permafrost project (2018-2021). The service comprises of the generation of remote sensing products for various regions and spatial scales as well as the specific infrastructures for visualisation, dissemination and access to datasets - PerSys. PerSys is the ESA GlobPermafrost geospatial information service for publishing and visualisation of information and data products to the public. Data products are described and searchable in the PerSys data catalogue (apgc.awi.de), and data visualisation employs the AWI WebGIS-infrastructure maps@awi (http://maps.awi.de), a highly scalable data visualisation unit within the AWI data-workflow framework O2A, from Observation to Archive. maps@awi WebGIS technology supports the project-specific visualisation of raster and vector data products of any spatial resolution and remote sensing origin. This is a prerequisite for the visualisation of the wide range of GlobPermafrost remote sensing products like: Landsat multispectral index trends (Tasseled Cap Brightness, Greeness, Wetness; Normalized Vegetation Index NDVI), Arctic land cover (e.g. shrub height, vegetation composition), lake ice grounding, InSAR-based land surface deformation, rock glacier velocities and a spatially distributed permafrost model output with permafrost probability and ground temperature per pixel. We established several WebGIS projects for the adaption to products specific spatial scales. For example, the WebGIS ‘Arctic’ visualises the Circum-Artic products. Highly resolved data products for rock glacier movements are visualised on regional scales in the WebGIS projects ‘Alps’, ‘Andes’ or ‘Central Asia’. The PerSYS WebGIS also visualises the stations of the WMO GCOS ground monitoring networks of the permafrost community: the Global Terrestrial Network for Permafrost GTN-P managed by the International Permafrost Association IPA. The PerSYS WebGIS has been continuously adapted in close co-operation with user at user workshops and at conferences and the International Permafrost Association (IPA)

O2A - Data Flow Framework from Sensor Observations to Archives

The Alfred Wegener Institute coordinates German polar research and is one of the most productive polar research institutions worldwide with scientists working in both Polar Regions – a task that can only be successful with the help of excellent infrastructure and logistics. Conducting research in the Arctic and Antarctic requires research stations staffed throughout the year as the basis for expeditions and data collection. It needs research vessels, aircrafts and long-term observatories for large-scale measurements as well as sophisticated technology. In this sense, the AWI also provides this infrastructure and competence to national and international partners. To meet the challenge the AWI has been progressively developing and sustaining an e-Infrastructure for coherent discovery, visualization, dissemination and archival of scientific information and data. Most of the data originates from research activities being carried out in a wide range of sea-, airand land-based operating research platforms. Archival and publishing in PANGAEA repository along with DOI assignment to individual datasets is a pursued end-of-line step. Within AWI, a workflow for data acquisition from vessel-mounted devices along with ingestion procedures for the raw data into the institutional archives has been well established. However, the increasing number of ocean-based stations and respective sensors along with heterogeneous project-driven requirements towards satellite communication, sensor monitoring, quality control and validation, processing algorithms, visualization and dissemination has recently lead us to build a more generic and cost-effective framework, hereafter named O2A (observations to archives). The main strengths of our framework (https://www.awi.de/en/data-flow) are the seamless flow of sensor observation to archives and the fact that it complies with internationally used OGC standards and assuring interoperability in international context (e.g. SOS/SWE, WMS, WFS, etc.). O2A comprises several extensible and exchangeable modules (e.g. controlled vocabularies and gazetteers, file type and structure validation, aggregation solutions, processing algorithms, etc.) as well as various interoperability services. We are providing integrated tools for standardized platform, device and sensor descriptions following SensorML (https://sensor.awi.de), automated near-real time and “big data” data streams supporting SOS and O&M and dashboards allowing data specialists to monitor their data streams for trends and early detection of malfunction of sensors (https://dashboard.awi.de). Also in the context of the “Helmholtz Data Federation” with outlook towards the European Open Science Cloud we are developing a cloud-based workspace providing user-friendly solutions for data storage on petabyte-scale and state-of-the-art computing solutions (Hadoop, Spark, Notebooks, rasdaman, etc.) to support scientists in collaborative data analysis and visualization activities including geo-information systems (http://maps.awi.de). Our affiliated repositories offer archival and long-term preservation as well as publication solutions for data, data products, publications, presentations and field reports (https://www.pangaea.de, https://epic.awi.de)

ESA GlobPermafrost - WebGIS based Visualisation of Remote Sensing Data

GIS server and desktop GIS technologies support scientific work at all levels, from data collection and data processing to data management and data visualisation. Here we present how the development and publication of scalable WebGIS data services supports the ESA DUE Globpermafrost (2016-2018), and the ESA CCI+ Permafrost (2018-2021) projects, specifically in the interaction with the permafrost community. Within ESA DUE programs, user feedback is essential to improve the remote sensing products. This is why ESA GlobPermafrost had to focus on methods and infrastructure for data presentation, and established PerSYS (Permafrost Information System). PerSYS became the ESA GlobPermafrost geospatial information service for publishing and visualisation of information and data products to the public. Data products are described and searchable in the PerSYS Data Catalogue, a core component of the Arctic Permafrost Geospatial Centre (APGC), established within the framework of ERC PETA-CARB at AWI. All GlobPermafrost data products will be DOI-registered and archived in the data archive PANGAEA provided by AWI. The data visualisation employs AWI’s WebGIS-infrastructure maps@awi (http://maps.awi.de), a highly scalable data visualisation unit within AWI’s data workflow framework O2A (from Observation to Archive). GIS services have been created and designed using ArcGIS for Desktop (latest Version) and finally published as a Web Map Service (WMS), an internationally standardized format (Open Geospatial Consortium (OGC)), using ArcGIS for Server. The project-specific data WMS as well as a resolution-specific background map WMS are embedded into a GIS viewer application based on Leaflet, an open-source JavaScript library. The GIS viewer application was adapted to interlink all GlobPermafrost WebGIS projects, and especially to enable their direct accessibility via the GlobPermafrost Overview WebGIS. The PerSys WebGIS is accessible via the GlobPermafrost project webpage and linked to the respective product groups as well as to maps@awi. WebGIS technology within maps@awi supports the project-specific visualisation of raster and vector data products of diverse spatial resolutions and remote sensing sources. This is a prerequisite for the visualisation of the wide range of GlobPermafrost remote sensing products like: Landsat multispectral index trends (Tasseled Cap Brightness, Greeness, Wetness; Normalized Vegetation Index NDVI), Arctic land cover (e.g., shrub height, vegetation composition), lake ice grounding, InSAR-based land surface deformation, rock glacier velocities and a spatially distributed permafrost model output with permafrost probability and ground temperature per pixel. All WebGIS projects are adapted to the products’ specific spatial scales. For example, the WebGIS ‘Arctic’ visualises the Circum-Artic products. Higher spatial resolution products for rock glacier movements are visualised on regional scales in the WebGIS projects ‘Alps’, ‘Andes’ or ‘Central Asia’. The PerSYS WebGIS also visualises the locations of the WMO GCOS ground monitoring networks of the permafrost community: the Global Terrestrial Network for Permafrost GTN-P managed by the International Permafrost Association IPA. The PerSYS WebGIS has been presented on several User workshops and at conferences, and is being continuously adapted in close interaction with the IPA

Tsunami-WebGIS - Displaying Tsunami Simulations for Indonesia to a Broader Audience

Within the German-Indonesian Tsunami Early Warning System project (GITEWS), a database containing approximately 4500 tsunami simulations for the Sunda arc, was created. Simulations were calculated using the finite element model TsunAWI implemented at AWI. The corresponding data products such as maximum wave height and estimated time of arrival at the coast are used operationally for tsunami early warning purposes. Visualization of tsunami simulation data in general and analyzing the information by non-modelers is nevertheless difficult, since accessing the data requires in depth knowledge on modeling with TsunAWI and specific programming libraries to be installed. However, it was intended to overcome this issue and provide an overview of the modeling efforts in Indonesia to a broader public. This was accomplished within the scope of the project Earth System Knowledge Platform (ESKP), initiated at institutions of the Helmholtz Association. Data products are visualized in the interactive Tsunami-WebGIS, a Geographical Information System (GIS) based web service, hosted on maps.awi.de, a GIS infrastructure implemented at AWI. The Tsunami-WebGIS provides an overview of the tsunami database and enables the user to trigger a tsunami at a certain epicenter and view the respective maximum wave heights as well as arrival time isochrones. This approach provides an overview of the simulation coverage and tsunami propagation in the the Sunda arc region to a non-expert audience. While it may be used for educational purposes, it has also proven to facilitate analysis by the modelers themselves. It is further planned to include extensions of the database in Indonesia as well as historical events in the region

PerSys - WebGIS-based permafrost data visualisation system for ESA GlobPermafrost

ESA DUE GlobPermafrost (www.globpermafrost.info) provides a remote sensing data service for permafrost research and applications. This service comprises of the generation of remote sensing products for various regions and spatial scales, and specific infrastructures for visualisation, dissemination and access to datasets. PerSys is the ESA GlobPermafrost geospatial information service for publishing and visualisation of information and data productstothepublic.DataproductsaredescribedandsearchableinthePerSysDataCatalogue,acorecomponent of the Arctic Permafrost Geospatial Centre (APGC), established within the framework of ERC PETA-CARB at AWI. The data visualisation employs the AWI WebGIS-infrastructure maps@awi (http://maps.awi.de), a highly scalable data visualisation unit within the AWI data-workflow framework O2A, from Observation to Archive. WebGIS technology in maps@awi supports the project-specific visualisation of raster and vector data products of diverse spatial resolutions and remote sensing sources. This is a prerequisite for the visualisation of the wide range of GlobPermafrost remote sensing products like: Landsat multispectral index trends (Tasseled Cap Brightness, Greeness, Wetness; Normalized Vegetation Index NDVI), Arctic land cover (e.g., shrub height, vegetation composition), lake ice grounding, InSAR-based land surface deformation, rock glacier velocities and a spatially distributed permafrost model output with permafrost probability and ground temperature per pixel. All WebGIS projects are adapted to the products specific spatial scale. For example, the WebGIS ‘Arctic’ visualises the Circum-Artic products. Higher spatial resolution products for rock glacier movements are visualised on regional scales in the WebGIS projects ‘Alps’, ‘Andes’ and ‘Central Asia’. GIS services were created and designed using ArcGIS for Desktop (10.4) and finally published as a Web MapService(WMS),aninternationallystandardizedformat(OpenGeospatialConsortium(OGC)),usingArcGIS for Server (10.4). The project-specific data WMS as well as a resolution-specific background map WMS are embedded into a GIS viewer application based on Leaflet, an open-source JavaScript library. The GIS viewer application was adapted to interlink all WebGIS projects, and especially to enable their direct accessibility via the GlobPermafrost Overview WebGIS project. The PerSys WebGIS is accessible via the GlobPermafrost project webpage and linked to the respective product groups as well as on maps@awi (maps.awi.de). All GlobPermafrost data products will be DOI-registered and archived in PANGAEA. In future, PerSys intends to encourage permafrost researchers other than GlobPermafrost to integrate and visualise their dat

SENSOR.awi.de: Management of heterogeneous platforms and sensors

SENSOR.awi.de is a component of our data flow framework designed to enable a semi-automated flow of sensor observations to archives (acronym O2A). The dramatic increase in the number and type of platforms and respective sensors operated by Alfred Wegener Institute along with complex project-driven requirements in terms of satellite communication, sensor monitoring, quality control and validation, processing pipelines, visualization, and archival under FAIR principles, led us to build a generic and cost-effective data flow framework. Most important, all components and services which make up this framework are extensible and exchangeble, were built using open access technologies (e.g. elastic search) and vocabularies (SeaVox NERC 2.0 vocabulary) and are compliant with various interoperability standards recommended by the international community. In this poster we illustrate the SENSOR.awi.de component which is the first step in the data acquisition process. In this component we have adopted the OGC standard SensorML 2.0 in oder to describe not only sensor-specific information (provenance/lineage metadata, data governance, physical characteristics, sensor positioning within the platform, parameter accuracy, etc) but also related events (e.g. station numbers as assigned in the data acquisition system on board along with actions such as deployment and recovery) and digital resources relevant for documenting the scientific workflows (e.g. calibration certificates). For this sake we have developed an AWI-specific SensorML profile and are sharing the model and as parters in the EU-funded project ODIP II we are contributing towards the generic Marine Sensor Profile. We have also been systematically sharing our experience in the RDA "Martina data Harmonization" Interest Group. In SENSOR.awi.de we are not only keen to describe sensors but also to create a sustainable identificaiton solution. Because the payload of various platforms change with time and sensor calibration may affect the data streams, it is important to keep track of these changes. For this sake we set up an audit trail history solution which allows scientists to create and identify an individual version/instance of a sensor. Each individual sensor instance and version gets assigned a handle as persistent identifier. These persistent identifiers enable the creation of citations for individual sensor instances at a given timestamp which can be used in publications and as part of the metadata associated with the final dataset and/or data product archived, e.g., in PANGAEA. To date, ~1300 sensor have been described in SENSOR.awi.de. Scientific Discipline/Research Area: Sensor Registry, Persistent Identification of Instruments, Provenance Metadata Relevance/Link to RDA: Input to/from RDA-Groups: "Persistent Identification of Instruments", "Data Citation" and "PID Kernel Information" Working Groups and "Vocabulary Services" Interest Group